COMMENT ⊗   VALID 00010 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.require "memo.pub[let,jmc]" source
C00003 00003	.bb INTRODUCTION
C00012 00004	.bb WHY ASCRIBE MENTAL QUALITIES?
C00022 00005	.bb TWO METHODS OF DEFINITION AND THEIR APPLICATION TO MENTAL QUALITIES
C00048 00006	.bb EXAMPLES OF SYSTEMS WITH MENTAL QUALITIES
C00081 00007	.bb |"GLOSSARY" OF MENTAL QUALITIES|
C00096 00008	.bb OTHER VIEWS ABOUT MIND
C00097 00009	.PORTION NOTES
C00106 00010	.bb REFERENCES
C00107 ENDMK
C⊗;
.require "memo.pub[let,jmc]" source;
.skip 15
.cb ASCRIBING MENTAL QUALITIES TO MACHINES

Abstract: Ascribing mental qualities like ⊗beliefs, ⊗intentions and
⊗wants to machines is correct and useful if done conservatively.
We propose some new definitional tools for this: second order
structural definitions and definitions relative to an approximate
theory.
.skip 5
(this draft of MENTAL[S76,JMC]@SU-AI compiled at {TIME} on {DATE})
.skip to column 1
.bb INTRODUCTION

	To ascribe certain %2beliefs%1, %2knowledge%1, %2free will%1,
%2intentions%1,  %2consciousness%1,  %2abilities%1  or %2wants%1 to a
machine or computer program is %3legitimate%1  when such an ascription
expresses  the  same  information  about  the machine that it expresses
about a person.     It  is  %3useful%1  when  the  ascription  helps  us
understand the structure of the machine, its past or future behavior,
or how to repair or improve it.  It is perhaps never %3logically
required%1 even for humans, but a practical theory of the behavior of
machines or humans may require mental qualities or qualities isomorphic to
them.  Theories of belief, knowledge and wanting can be constructed for
machines in a simpler setting than for humans and later applied to humans.
Ascription of mental qualities is %3most straightforward%1 for machines of
known structure such as thermostats and computer operating systems, but is
%3most useful%1 when applied to entities whose structure is very
incompletely known.

	The above  views are motivated by work in  artificial intelligence$
(abbreviated  AI).  They can  be taken as asserting that
many of the philosophical problems  of mind take a practical form  as
soon  as one  takes  seriously  the idea  of  making machines  behave
intelligently.  In particular, AI raises for machines two issues that
have heretofore been considered only in connection with people.

	First, in designing intelligent programs  and looking at them
from  the outside  we need  to determine  the conditions  under which
specific mental and volitional terms  are applicable.  We can  exemplify
these  problems by  asking  when might it be legitimate  to  say about  a
machine, %2" It knows I want a reservation to Boston, and it can give
it to me, but it won't"%1.

~Work in artificial intelligence is still far from showing
how to reach human-level intellectual performance.
Our approach to the AI problem involves identifying the intellectual
mechanisms required for problem solving and describing them precisely.
Therefore we are at the end of the philosophical
spectrum that requires everything to be formalized in mathematical logic.
It  is sometimes  said  that  one
studies philosophy  in order to advance beyond  one's untutored naive
world-view, but unfortunately for  artificial intelligence, no-one has
yet been able to give a description of even a naive world-view,
complete and precise enough to allow a knowledge-seeking program to
be constructed in accordance with its tenets.
~
	Second, when we  want a %3generally intelligent%1$ computer
program, we must build into it a %3general view%1 of what the world
is like with  especial attention to facts  about how the  information
required to solve problems is to be  obtained and used.  Thus we must
provide   it   with   some   kind   of   %2metaphysics%1 (general
world-view)  and %2epistemology%1 (theory of knowledge) however naive.

~Present AI programs operate in limited domains, e.g. play particular
games, prove theorems in a particular logical system, or understand
natural language sentences covering a particular subject matter and
with other semantic restrictions.  General intelligence will require
general models of situations changing in time, actors with goals
and strategies for achieving them, and knowledge about how information
can be obtained.~

	As  much  as  possible,  we  will  ascribe  mental  qualities
separately from  each other instead of bundling  them  in a
concept of mind.   This is necessary,  because present machines  have
rather   varied  little   minds;  the   mental  qualities   that  can
legitimately  be ascribed to them are few  and differ from machine to
machine.   We  will  not  even try  to  meet  objections  like,
%2"Unless it also  does X, it is illegitimate to  speak of its having
mental qualities."%1

	Machines  as  simple  as thermostats  can  be  said  to  have
beliefs,  and having  beliefs seems  to be  a characteristic  of most
machines  capable of  problem  solving  performance.    However,  the
machines mankind has so far found  it useful to construct rarely have
beliefs  about beliefs.    (Beliefs about  beliefs will be  needed by
computer programs to reason about what  knowledge they
lack and where to  get it).  Mental qualities  peculiar to human-like
motivational structures, such  as love and hate, will not be required
for intelligent behavior, but we could probably program  computers to
exhibit them if we wanted to,  because our common sense notions about
them  translate  readily into  certain  program and  data structures.
Still other mental qualities, e.g.  humor and appreciation of beauty,
seem  much harder  to  model.   While  we will  be  quite liberal  in
ascribing ⊗some mental qualities  even to rather primitive  machines,
we will  try to  be conservative  in our  criteria for ascribing  any
⊗particular quality.

	The successive sections of this paper will give philosophical
and AI reasons for ascribing beliefs to machines, two new forms of
definition that seem necessary for defining mental qualities and
examples of their use, examples of systems to which mental
qualities are ascribed, some first attempts at defining a variety
of mental qualities, some criticisms of other views on mental
qualities, notes, and references.

.SKIP TO COLUMN 1
.ONCE CENTER
.bb WHY ASCRIBE MENTAL QUALITIES?


	%3Why  should we want to  ascribe beliefs to  machines  at all?%1
This is the opposite question to that of
%2reductionism%1.  Instead of asking how mental qualities can be
%3reduced%1 to physical ones, we ask how to %3ascribe%1 mental qualities
to physical systems.  This question may be more straightforward
and may lead to better answers to the questions of reductionism.

	To put  the issue  sharply, consider  a computer program  for
which  we possess complete listings.   The behavior of  the program in
any environment is determined from  the structure of the program  and
can be  found out  by simulating  the action of  the program  and the
environment  without  having  to deal  with  any  concept  of belief.
Nevertheless, there  are  several reasons  for  ascribing belief  and
other mental qualities:

.ITEM←0;
	#. Although we may know the program, its state at a given moment
is usually not directly observable, and the facts we can obtain
about its current state may be more readily expressed by ascribing
certain beliefs or wants than in any other way.

	#. Even if we can simulate the interaction of our program
with its environment using another more comprehensive program, the
simulation may be a billion times too slow.  We also
may not have the initial conditions of the environment or the environment's
laws of
motion in  a suitable  form, whereas  it may  be feasible  to make  a
prediction of  the effects of the  beliefs we ascribe to  the program
without any computer at all.

	#. Ascribing beliefs may allow deriving general statements
about the program's behavior that could not be obtained from any
finite number of simulations.

	#.   The belief and goal structures we ascribe to the program
may be easier to understand than the details of  program as expressed
in its listing.

	#. The belief and goal structure is likely to be close to the
structure the designer  of the  program had in  mind, and  it may  be
easier to debug the program in terms  of this structure than directly
from  the listing.   In  fact, it  is often  possible for  someone to
correct a fault by reasoning  in general terms about the  information
in a program or machine, diagnosing  what is wrong as a false belief,
and   looking  at  the  details  of   the  program  or  machine  only
sufficiently to determine how  the false belief is represented  and
what mechanism caused it to arise.$

	All the above reasons for ascribing beliefs are epistemological.
i.e. ascribing beliefs is needed to adapt to limitations on our
ability to acquire knowledge, use it for prediction, and establish
generalizations in terms of the elementary structure of the program.
Perhaps this is the general reason for ascribing higher levels of
organization to systems.

~	This  kind  of  teleological  analysis  is  often  useful  in
understanding natural organisms as well as machines.  Here  evolution
takes  the place  of  design and  we  often understand  the  function
performed by  an organ before we  understand its detailed physiology.
Teleological analysis  is  applicable to  psychological  and  social
phenomena in  so far as  these are designed  or have been  subject to
selection.

	However,  teleological analysis fails when applied to aspects
of nature which  have neither been  designed nor produced by  natural
selection from a population.   Much medieval science was based on the
Judeo-Christian-Moslem  hypothesis that  the details of  the
world were designed by God  for the benefit of man.   The strong form
of  this hypothesis was abandoned  at the time of  Galileo and Newton
but occasionally recurs.   Barry Commoner's (1972) axiom of  ecology
"Nature knows best"  seems to be mistakenly based on  the notion that
nature  as a  whole  is the  result of  an evolutionary  process that
selected the "best nature".~

	Computers give rise to numerous examples of building a higher
structure on the basis of a lower and conducting subsequent analyses
using the higher structure.  The geometry of the electric fields in
a transistor and its chemical composition give rise to its properties
as an electric circuit element.
Transistors are combined in small circuits and powered in standard
ways to make logical elements such as ANDs, ORs, NOTs and flip-flops.
Computers are designed with these logical elements
to obey a desired order code; the designer usually needn't
consider the properties of the transistors as circuit elements.
The designer of a higher level language works with the order
code and doesn't have to know about the ANDs and ORs; the user of
the higher order language needn't know the computer's order code.

	In the above cases, users of the  higher level can completely
ignore  the lower level, because the  behavior of the higher  level system is
completely determined by  the values of  the higher level  variables;
e.g.   in order to determine  the outcome of a  computer program, one
needn't consider the flip-flops.  However, when
we ascribe mental structure to humans or goals to  society, we always
get highly  incomplete systems; the  higher level behavior  cannot be
fully predicted  from  higher  level observations  and  higher  level
"laws" even when the underlying lower level behavior is determinate.

	Besides the above philosophical reasons for ascribing mental
qualities to machines, I shall argue that in order to make machines
behave intelligently, we will have to program them to ascribe beliefs
etc. to each other and to people.

→→→→Here there will be more on machine's models of each others minds.←←←←


.SKIP TO COLUMN 1
.bb TWO METHODS OF DEFINITION AND THEIR APPLICATION TO MENTAL QUALITIES

	In our opinion, a major source of problems in defining mental
and other philosophical concepts is the weakness of the methods of
definition that have been %2explicitly%1 used.  Therefore we introduce two
new$ kinds of definition: %2second order structural definition%1 and
%2definition relative to an approximate theory%1 and their application
to defining mental qualities.
~Novelty is not absolutely guaranteed.~

.ITEM←0;

#. %3Second Order Structural Definition.%1

	Structural definitions of qualities are given in terms of the
state of the  system being described while behavioral definitions are
given in terms of its actual or potential behavior$.

	~Behavioral  definitions are often  favored in  philosophy.  A
system is  defined to  have  a certain  quality if  it behaves  in  a
certain way  or  is  %2disposed%1 to behave  in a  certain way.
Their  ostensible  virtue  is  conservatism;   they  don't  postulate
internal  states that  are unobservable  to  present science  and may
remain unobservable.
However, such  definitions are awkward for  mental qualities, because,
as common sense  suggests, a  mental quality  may  not  result  in
behavior, because another mental  quality may prevent it; e.g.  I may
think you are  thick-headed, but politeness may prevent my saying so.
Particular  difficulties  can  be  overcome,  but  an  impression  of
vagueness remains.  The liking  for behavioral definitions stems from
caution, but  I would interpret scientific experience as showing that
boldness in postulating  complex structures of unobserved  entities -
provided it is  accompanied by a willingness to  take back mistakes -
is more likely to be rewarded by understanding of  and  control  over
nature   than   is   positivistic   timidity.    It  is  particularly
instructive to imagine a determined behaviorist trying to figure  out
an  electronic  computer.  Trying to define each quality behaviorally
would get him nowhere;  only  simultaneously  postulating  a  complex
structure  including  memory, arithmetic unit, control structure, and
input-output would yield predictions  that  could  be  compared  with
experiment.

	There is a sense in which operational definitions are not taken
seriously even by their proposers.  Suppose someone gives
an operational definition of length (e.g. involving a certain platinum bar),
and a whole school of physicists and philosophers
becomes quite attached to it.  A few years later, someone else criticizes
the definition as lacking some desirable property, proposes a change,
and the change is accepted.  This is normal, but
if the original definition expressed what they really meant by
the length, they would refuse to change, arguing that the new concept
may have its uses, but it isn't what they mean by "length".  This shows
that the concept of "length" as a property of objects is more stable
than any operational definition.

	Carnap has an interesting section in %2Meaning and Necessity%1
entitled "The Concept of Intension for a Robot" in which he
makes a similar point saying, %2"It is clear that the method
of structural analysis, if applicable, is more powerful than
the behavioristic method, because it can supply a general
answer, and, under favorable circumstances, even a complete
answer to the question of the intension of a given predicate."%1
~

	If the structure of the machine is known, one can give an ad hoc
%2first  order  structural  definition%1.    This is a predicate ⊗B(s,p) 
where ⊗s represents a state of the machine and ⊗p represents a sentence
in a suitable language, and ⊗B(s,p) is the assertion that when the
machine is in state ⊗s, it ⊗believes the sentence ⊗p.  (Note that
a sentence is the object believed rather than taking belief as a modal
operator).
A general ⊗first ⊗order structural definition of belief would be a
predicate ⊗B(W,M,s,p) where ⊗W is the "world" in which the machine
⊗M whose beliefs are in question is situated.
I do not see how to give such a definition of belief, and
I think it is impossible.
Therefore we turn to second order definitions.

	A second order structural definition of belief is a second
order predicate ⊗β(W,M,B).  ⊗β(W,M,B) asserts that the first order
predicate ⊗B is a "good" notion of belief for the machine ⊗M in
the world ⊗W.  Here "good" means that the beliefs that ⊗B ascribes
to ⊗M agree with our ideas of what beliefs ⊗M would have, not that
the beliefs themselves are true.
The axiomatizations of belief in the literature are partial
second order definitions.

	In general, %3a second order definition gives criteria for
criticizing an ascription of a quality to a system.%1
We suggest that both our common sense and scientific usage of
not-directly-observable qualities corresponds more losely to second
order structural definition than to any kind of behavioral definition.
Note that  a  second order  definition
cannot  guarantee  that  there exist  predicates  %2B%1  meeting the
criterion β or that such a %2B%1 is unique.
Some qualities are best defined jointly with related
qualities, e.g. beliefs and goals may require joint treatment.

	Second   order   definitions criticize   whole   belief
structures rather than  individual beliefs.  We can  treat individual
beliefs  by  saying  that a  system  believes  %2p%1  in state  %2s%1
provided all "reasonably good"  %2B%1's satisfy %2B(s,p)%1.  Thus  we
are distinguishing the "intersection" of the reasonably good %2B%1's.

	(An analogy with cryptography may be helpful.  
We solve a cryptogram by making hypotheses about the structure of the
cipher and about the translation of parts of the cipher text.  Our
solution is complete when we have "guessed" a cipher system that produces
the cryptogram from a plausible plaintext message.  Though we never
prove that our solution is unique, two different solutions are
almost never found except for very short cryptograms.  In the analogy,
the second order definition β corresponds to the general idea of
encipherment, and ⊗B is the particular system used.  While we will
rarely be able to prove uniqueness, we don't expect to find two %2B%1s
both satisfying β).

	It  seems  to  me  that  there should  be  a  metatheorem  of
mathematical  logic asserting  that not all  second order definitions
can  be reduced  to  first  order definitions  and  further  theorems
characterizing  those  second   order  definitions  that  admit  such
reductions.  Such  technical results, if  they can be  found, may  be
helpful in  philosophy and in  the construction of  formal scientific
theories.  I would conjecture that many of the informal philosophical
arguments that certain mental  concepts cannot be reduced to  physics
will turn out to be sketches of arguments that these concepts require
second (or higher) order definitions.

	Here  is a deliberately imprecise second order definition of
belief.  For each state %2s%1 of the  machine and
each  sentence ⊗p in  a suitable  language ⊗L,
we assign truth to %2B(s,p)%1 if and only if
the machine is considered to believe %2p%1  when
it  is  in state  %2s%1.    The  language  %2L%1 is  chosen  for  our
convenience, and  there is no assumption  that the machine explicitly
represents sentences of %2L%1 in any way.  Thus we can talk about the
beliefs of  Chinese,  dogs, corporations,  thermostats, and  computer
operating  systems  without assuming  that  they use  English  or our
favorite first order language.  ⊗L may or may not be the  language be
the language we are using for making other assertions, e.g. we could,
writing in English, systematically use French sentences as objects of
belief.  However,  the best choice  for artificial intelligence  work
may be to  make ⊗L a subset of our  "outer" language restricted so as
to avoid the paradoxical self-references of (Montague 1963).

	We now subject ⊗B(s,p) to certain criteria; i.e. β⊗(B,W) 
is considered true provided the following conditions are satisfied:

	&. The set %2Bel(s)%1 of beliefs, i.e. the set of %2p%1's for
which  %2B(s,p)%1 is  assigned true  when ⊗M is in state ⊗s 
contains  sufficiently "obvious"
consequences of some of its members.

	&.   %2Bel(s)%1 changes in a  reasonable  way when the  state
changes in time.    We   like  new  beliefs  to  be  logical  or  "plausible"
consequences of old ones or to come in as %2communications%1  in some
language on the input lines or  to be %2observations%1, i.e.  beliefs
about the environment the information for which comes in on the input
lines.  The set of beliefs should not change too rapidly as the state
changes with time.

	&.   We prefer  the set  of beliefs  to be  as consistent  as
possible.  (Admittedly, consistency  is not a quantitative concept in
mathematical logic -  a system is  either consistent or  not, but  it
would seem that we  will sometimes have to ascribe  inconsistent sets
of beliefs to machines and people.  Our intuition says that we should
be able to maintain areas of  consistency in our beliefs and that  it
may be  especially important  to avoid  inconsistencies in  the machine's
purely analytic beliefs).

	&.  Our criteria for belief systems can be strengthened if we
identify some of the  machine's beliefs as expressing goals,  i.e. if
we have beliefs  of the form "It would be good if  ...".  Then we can
ask that the machine's behavior  be somewhat %2rational%1, i.e.  %2it
does what  it  believes will  achieve its  goals%1. The  more of  its
behavior  we can  account for in  this way,  the better we  will like
%2B(s,p)%1.  We also would like to account for internal state changes
as changes in belief in so far as this is reasonable.

	&.  If the machine communicates, i.e. emits sentences in some
language  that  can  be  interpreted  as  assertions,  questions  and
commands, we will want the assertions to be among  its beliefs unless
we are  ascribing to it  a goal or  subgoal that involves  lying.  In
general, its  communications  should  be such  as  it  believes  will
achieve its goals.

	&.  Sometimes we shall want to ascribe introspective beliefs,
e.g. a belief that it does not know how to fly to Boston or even that
it doesn't know what it wants in a certain situation.

	&. Finally, we will prefer a more economical ascription %2B%1
to a less economical one.  The  fewer beliefs we ascribe and the less
they change with state consistent with accounting for the
behavior and the internal state changes, the better we will like it.

	The above  criteria have  been  formulated somewhat  vaguely.
This  would be bad if there  were widely different ascriptions of  beliefs
to a particular machine that all met our  criteria or if the criteria
allowed  ascriptions that  differed widely from  our intuitions.   My
present opinion is that more thought will make the  criteria somewhat
more precise  at no cost  in applicability, but that  they %2should%1
still remain  rather vague, i.e. we shall want to ascribe belief in a
%2family%1 of cases.
However, even at the present level of vagueness, there probably
won't be radically different equally "good" ascriptions of belief for
systems of practical interest.  If there were, we would notice
unresolvable ambiguities in our ascriptions of belief to our acquaintances.

	While we may not want to pin down our general idea of belief
to a single axiomatization, we will need to build precise axiomatizations of
belief and other mental qualities into particular intelligent computer
programs.


#. %3Definitions relative to an approximate theory%1.


	Certain  concepts,  e.g. %2X can do Y%1, are meaningful only in
connection with a rather complex theory.   For  example,  suppose  we
denote the state of the world by %2s%1, and suppose we have functions
%2f%41%2(s)%1,...,%2f%4n%2(s)%1  that  are  directly  or   indirectly
observable.  Suppose further that %2F(s)%1 is another function of the
world-state  but  that  we   can   approximate   it by

	%2F"(s) = F'(f%41%2(s),...,f%4n%2(s))%1.

	Now  consider  the counterfactual
conditional sentence, "If %2f%42%2(s)%1 were 4, then %2F(s)%1 would be
3 - calling the present state of the world %2s%40%1." By itself, this
sentence has no meaning, because no definite state %2s%1 of the world
is  specified  by  the  condition.   However, in the framework of the
functions %2f%41%2(s),...,f%4n%2(s)%1 and the given approximation  to
%2F(s)%1,  the  assertion  can be verified by computing ⊗F' with all
arguments except the second having the  values  associated  with  the
state %2s%40%1 of the world.

	This gives rise to some remarks:

&. The most straightforward case of counterfactuals arises when
the state of a phenomenon has a distinguished Cartesian product
structure.  Then the meaning of a change of one component without
changing the others is quite clear.  Changes of more than one
component also have definite meanings.  This is a stronger
structure than the %2possible worlds%1 structure discussed in
(Lewis 1973).

&. The usual case is one in which the state %2s%1 is a substantially
unknown entity and the form of the function %2F%1 is also
unknown, but the values of %2f%41%2(s),...,f%4n%2(s)%1 and
the function %2F'%1 are much better known.
Suppose further that %2F"(s)%1 is known to be only a fair approximation
to %2F(s)%1.  We now have a situation in which the counterfactual
conditional statement is meaningful as long as it is not examined too
closely, i.e. as long as we are thinking of the world in terms of
the values of %2f%41%2,...,f%4n%1, but when we go beyond the
approximate theory, the whole meaning of the sentence seems to
disintegrate.

	Our idea is that this is a very common phenomenon. In
particular it applies to statements of the form %2"X can do Y"%1.
Such statements can be given a precise meaning in terms
of a system of interacting automata as is discussed in detail in
(McCarthy and Hayes 1970).  We determine whether Automaton 1 can put Automaton
3 in state 5 at time 10 by answering a question about an automaton system
in which the outputs from Automaton 1 are replaced by inputs from
outside the system.  Namely, we ask whether there is a sequence of
inputs to the new system that ⊗would put Automaton 3 in state 5 at time 10;
if yes, we say that Automaton 1 ⊗could do it in the original system
even though we may be able to show that it won't emit the necessary
outputs.  In that paper, we argue that this definition corresponds
to the intuitive notion of %2X can do Y.%1.

	What was not noted in that paper is that modelling the
situation by the particular system of interacting automata is
an approximation, and the sentences involving
⊗can derived from the approximation
cannot necessarily be translated into single assertions about the
real world. 

	I contend  that the statement,  %2"I can go  skiing tomorrow,
but I don't intend to, because I want to finish this paper"%1 has the
following properties:

	1. It has a precise meaning in a certain approximate theory of
the world
in which I and my environment are considered as collections of interacting
automata.

	2. It cannot be directly interpreted as a statement about the world
itself, because it can't be stated in what total configurations of the
world the success of my attempt to go skiing is to be validated.

	3. The approximate theory within which the statement is meaningful
may have an objectively preferred status in that it may be the only theory
not enormously more complex that enables my actions and mental states to
be predicted.

	4. The statement may convey useful information.

Our conclusion is that the statement is %3true%1, but in a sense that
depends essentially on the approximate theory, and that this intellectual
situation is normal and should be accepted.  We further conclude that
the old-fashioned common-sense analysis of a personality into %2will%1
and %2intellect%1 and other components may be valid and might be put
on a precise scientific footing using %2definitions relative to an
approximate theory%1.


.SKIP TO COLUMN 1
.ONCE CENTER
.bb EXAMPLES OF SYSTEMS WITH MENTAL QUALITIES

	Let us  consider some  examples of machines  and programs  to
which we may ascribe belief and goal structures.

.ITEM←0;
	#.  %3Thermostats.%1   Ascribing beliefs to simple thermostats is
unnecessary for the study of thermostats,
because their operation can be well  understood
without it.  However, their very simplicity  makes it clearer what is
involved  in the  ascription, and  we maintain (partly  as a
provocation to  those who  regard  attribution of  beliefs  to
machines  as mere  intellectual  sloppiness) that  the ascription  is
legitimate.$

~Whether a system has beliefs and other mental qualities is not
primarily a matter of complexity of the system.  Although cars are
more complex than thermostats, it is hard to ascribe beliefs or
goals to them, and the same is perhaps true of the basic hardware of a
computer, i.e. the part of the computer that executes the program
without the program itself.
~
	First  consider a simple  thermostat that turns off the
heat when the  temperature is a degree  above the temperature set  on
the thermostat,  turns on the heat  when the temperature  is a degree
below the desired  temperature, and leaves  the heat as  is when  the
temperature  is   in  the  two   degree  range  around   the  desired
temperature. The simplest belief predicate %2B(s,p)%1 ascribes belief
to only three sentences:  "The room is too cold", "The room is  too
hot", and "The room is OK" - the beliefs being assigned to states
of  the thermostat in the obvious way.
When the
thermostat believes  the  room is  too cold  or too  hot, it sends  a
message saying so to the furnace. A slightly more complex belief
predicate could also  be used in  which the thermostat  has a  belief
about what the temperature should be and another belief about what it
is.   It is not clear  which is better, but if  we wished to consider
possible errors in the thermometer, then we would ascribe
beliefs  about what the temperature is. We  do not ascribe
to  it any other  beliefs; it has  no opinion even  about whether the
heat is on or off or about the weather or about who won the battle of
Waterloo.    Moreover, it  has  no introspective  beliefs,  i.e.   it
doesn't believe that it believes the room is too hot.

	The temperature control system in  my house may be  described
as  follows: Thermostats  upstairs and  downstairs  tell the  central
system  to turn  on or shut  off hot  water flow  to these areas.   A
central water-temperature thermostat tells the furnace to turn  on or
off  thus  keeping the  central  hot  water  reservoir at  the  right
temperture.  Recently it was too hot upstairs, and the question arose
as to whether the upstairs thermostat mistakenly  %2believed%1 it was
too  cold  upstairs  or  whether  the furnace  thermostat  mistakenly
%2believed %1 the  water was too  cold.  It  turned out that  neither
mistake was made; the downstairs controller %2tried%1 to turn off the
flow  of water but  %2couldn't%1, because the  valve was  stuck.  The
plumber came  once  and found  the trouble,  and  came again  when  a
replacement valve  was ordered.   Since the services of  plumbers are
increasingly  expensive, and  microcomputers are  increasingly cheap,
one is led to design a temperature control system that would %2know%1
a lot more about the thermal  state of the house and its own state of
health.

	In the first  place, while the  system %2couldn't%1 turn  off
the flow of hot water  upstairs, there is no reason  to ascribe to
it the  %2knowledge%1 that it couldn't, and  %2a fortiori%1 it had no
ability to %2communicate%1 this %2fact%1  or to take it into  account
in controlling the system.  A more advanced system would know whether
the  %2actions%1 it %2attempted%1 succeeded, and it would communicate
failures and adapt  to them.  (We  adapted to the failure  by turning
off  the whole  system  until the  whole  house cooled  off and  then
letting the two parts warm up  together.  The present system has  the
%2physical  capability%1  of  doing  this  even   if  it  hasn't  the
%2knowledge%1 or the %2will%1.
.skip 2

	#. %3Self-reproducing  intelligent  configurations  in  a  cellular
automaton world.%1  A ⊗cellular ⊗automaton system assigns to each vertex
in  a  certain graph a finite automaton.  The state of each automaton
at time ⊗t+1 depends on its state at time %2t%1 and the states  of
its  neighbors  at  time %2t%1. The most common graph is the array of
points ⊗(x,y) in the plane with integer co-ordinates ⊗x and ⊗y.   The
first  use of cellular automata was by von Neumann (196?) who found a
27 state automaton that could be used to  construct  self-reproducing
configuration   that  were  also  universal  computers.    The  basic
automaton in von Neumann's system had a distinguished state called  0
and  a  point in state 0 whose four neighbors were also in that state
would remain in state 0.  The initial configurations  considered  had
all  but  a  finite  number of cells in state 0, and, of course, this
property would persist although the number of  non-zero  cells  might
grow indefinitely with time.

	The  self-reproducing  system used the states of a long strip
of non-zero cells as a "tape" containing instructions to a "universal
constructor"  configuration  that  would  construct  a  copy  of  the
configuration to be reproduced but with each cell in a passive  state
that  would  persist  as  long  as its neighbors were also in passive
states.  After the construction phase, the tape would  be  copied  to
make  the  tape for the new machine, and then the new system would be
set in motion by activating one of its cells. The  new  system  would
then move away from its mother and the process would start over.  The
purpose of the design was to  demonstrate  that  arbitrarily  complex
configurations  could  be  self-reproducing  -  the  complexity being
assured by also requiring that they be universal computers.

	Since von Neumann's time, simpler basic cells admitting
self-reproducing universal computers have been discovered.
The simplest so far is the two state Life automaton of John Conway (196?).
The state of a cell at time ⊗t+1 is determined its state at time ⊗t 
and the states of its eight neighbors at time ⊗t.  Namely,
a point whose state is 0 will change to state 1 if exactly three
of its neighbors are in state 1.  A point whose state is 1 will
remain in state 1 if two or three of its neighbors are in state 1.
In all other cases the state becomes or remains 0.

	Conway's initial intent was to model a birth and death
process whereby a cell is born (goes into state 1) if it has the
right number of living neighbors (namely three) and dies if it
is either too lonely (has none or one neighbor in state 1) or is
overcrowded (has four or more living neighbors).  He also asked
whether infinitely growing configurations were possible, and
Gosper first proved that there were.
Surprisingly, it turned out that self-reproducing universal
computers could be built up as Life configurations.

	Consider a number of such
self-reproducing universal computers operating in the Life
plane, and suppose that they have been programmed to study
the properties of their world and to communicate among themselves
about it, perhaps pursuing various goals co-operatively and competitively.
Call these configurations robots.
In some respects their intellectual and scientific problems
will be like ours, but in one major respect they live
in a simpler world than ours seems to be.  Namely, 
the fundamental physics of their world is that of the life
automaton, and there is no obstacle to each robot ⊗knowing 
this physics, and being able to simulate
the evolution of a life configuration given the initial
state.  Moreover, if the initial state of the robot world is finite it can
have been recorded in each robot in the beginning or else
recorded on a strip of cells that the robots can read.
(The infinite regress of having to describe the description
is avoided by a convention that the description is not
separately described, but can be read ⊗both as a description of the
world ⊗and as a description of itself.)

	Since these robots know the initial state of their world
and its laws of motion, they can simulate as much
of their world's history as they want, assuming that each
can grow into unoccupied space so as to have memory to store
the states of the world being simulated.  This simulation
is necessarily much slower than real time, so they can never catch up with the
present - let alone predict the future.  This is clear if we
imagine the simulation carried out in a straightforwardly by updating
a list of currently active cells in the simulated world
according to the Life rule, but it also applies to any clever mathematical
method that might predict millions of steps ahead.  (Some Life
configurations, e.g. static ones or ones containing single ⊗gliders 
or ⊗cannon can have their distant futures predicted with little
computing.)  Namely, if there were an algorithm for such prediction,
a robot could be made that would predict its own future and then
disobey the prediction.  The detailed proof would be analogous to
the proof of unsolvability of the halting problem for Turing machines.

	Now we come to the point of this long disquisition.  Suppose
we wish to program a robot to be successful in the Life world in
competition or co-operation with the others.  Without any idea
of how to give a mathematical proof, I will claim that our robot
will need programs that ascribe purposes and beliefs to its fellow robots
and  predict how they will react to its  own
actions by assuming that %2they will act in ways that they believe
will achieve their goals%1.  Our robot might acquire these mental theories
in several ways:  First, we might design the universal machine so that
they are present in the initial configuration of the world.  Second, we might
program it to acquire this ideas by induction from its experience
and even transmit them to others through an "educational system".  Third,
it might derive the psychological laws from the fundamental physics of
the world and its knowledge of the initial configuration,
and finally, it might discover how robots are built from Life cells by doing
experimental "biology".

	Knowing the Life  physics without some information  about the
initial configuration is insufficient to derive the %2psychological%1
laws, because  robots can  be constructed  in the  Life  world in  an
infinity of ways.  This follows from the "folk theorem" that the Life
automaton  is universal in the sense  that any cellular automaton can
be constructed by taking sufficiently large squares of  Life cells as
the basic cell of the other automaton.$

	Our own intellectual position is more difficult than that of
the Life robots.  We don't know the fundamental physics of our world,
and we can't even be sure that its fundamental physics is describable
in finite terms.  Even if we knew the physical laws, they seem to
preclude precise knowledge of an initial state and precise calculation
of its future both for quantum mechanical reasons and because the
continuous functions needed to represent fields can't seem to involve
an in inite amount of information.

~ Our own ability to derive the laws of higher levels of organization
from knowledge  of lower level laws is  also limited by universality.
While there appears to be essentially one possible  chemistry allowed
by the laws of physics, the  laws of physics and chemistry allow many
biologies, and,  because the neuron is a universal computing element,
an arbitrary  mental structure is  allowed by basic  neurophysiology.
Therefore,  to  determine  human  mental  structure,  one  must  make
psychological  experiments,  ⊗or  determine  the  actual   anatomical
structure of the  brain and the  information stored in it .
One cannot determine the structure of the brain merely
from  the fact that the  brain is capable of  certain problem solving
performance.  In this  respect, our position
is similar to that of the Life robot.~

	One point of the cellular automaton robot  example is to make
plausible  the idea  that much  of human mental  structure is  not an
accident of evolution  or even of  the physics of  our world, but  is
required for successful problem solving behavior and must be designed
into or evolved by any system that exhibits such behavior.
.skip 2

	#. %3Computer time-sharing systems.%1  These complicated
computer programs allocate computer time and other resources among users.
They allow each user of the computer to behave as though he had
a computer of his own, but also allow them to share files of
data and programs and to communicate with each other.
They are often used for many years with continual small changes, and
and the people making the changes and correcting errors
are often different not the original authors of the system.
A person confronted with the task of correcting a malfunction or
making a change in a time-sharing system can conveniently use a
mentalistic model of the system.

	Thus suppose a user complains that the system
will not run his program.  Perhaps the system believes that he
doesn't want to run, perhaps it persistently believes that he
has just run, perhaps it believes that his quota of computer
resources is exhausted, or perhaps it believes that his program
requires a resource that is unavailable.  Testing these hypotheses
can often be done with surprisingly little understanding of the
internal workings of the program.

→→→→→There will be more examples here of the belief of time-sharing systems.←←←
.skip 2
	#. %3Programs designed to reason.%1  Suppose we explicitly design a
program to represent information by sentences in a certain language
stored in the memory of the computer and decide what to do by making
inferences, and doing what it concludes will advance its goals.  Naturally,
we would hope that our previous second order definition of belief will
"approve of" a %2B(p,s)%1 that ascribed to the program believing the
sentences explicitly built in.  We would be somewhat embarassed if
someone were to show that our second order definition approved as
well or better of an entirely different set of beliefs.

	Such a program was first proposed in (McCarthy 1960), and here is how
it might work:

	Information about the world is stored in a wide variety of
data structures.  For example, a visual scene received by a TV
camera may be represented by a 512x512x3 array of numbers representing
the intensities of three colors at the points of the visual field.
At another level, the same scene may be represented by a list of regions,
and at a further level there may be a list of physical objects and their
parts together with other information about these objects obtained from
non-visual sources.  Moreover, information about how to solve various
kinds of problems may be represented by programs in some programming
language.

	However, all the above representations are subordinate to
a collection of sentences in a suitable first order language that
includes set theory.  By subordinate, we mean that there are sentences
that tell what the data structures represent and what the programs do.
New sentences can arise by a variety of processes: inference from
sentences already present, by computation from the data structures representing
observations, ...

→→→→→There will be more here about what mental qualities should be programmed.←←←

.SKIP TO COLUMN 1
.ONCE CENTER
.bb |"GLOSSARY" OF MENTAL QUALITIES|

	In this section we give short "definitions" for machines of a collection
of mental qualities.  We include a number of terms which give us
difficulty with an indication of what the difficulties seem to be.

.ITEM←0;

	#. %3Actions%1.  We want to distinguish the actions of a being
from events that occur in its body and that affect the outside
world.  For example, we wish to distinguish a random twitch
from a purposeful movement.  This is not difficult %2relative
to a theory of belief that includes intentions%1.  One's purposeful
actions are those that would have been different had one's intentions
been different.  This requires that the theory of belief have sufficient
Cartesian product structure so that the counterfactual conditional
`"if its intentions had been different" is defined in the theory.
As explained in the section on definitions relative to an approximate
theory, it is not necessary that the counterfactual be given a
meaning in terms of the real world.

	#. %3Introspection and self-knowledge.%1

	We  say that  a  machine introspects  when it  comes  to have
beliefs about its own mental  state.  A simple form of  introspection
takes  place  when  a  program  determines  whether  it  has  certain
information  and if not asks for it.   Often an operating system will
compute a check  sum of itself  every few minutes  to verify that  it
hasn't been changed by a software or hardware malfunction.

	In principle,  introspection is easier  for computer programs
than for people, because the entire memory in which programs and data
are stored is available for inspection.   In fact, a computer program
can  be  made to  predict how  it  would react  to  particular inputs
provided it has enough free storage to perform the calculation.  This
situation smells of paradox, and there  is one.  Namely, if a program
could predict its own actions in less time than it takes to carry out
the action, it could refuse  to do what it has predicted  for itself.
This only  shows that self-simulation  is necessarily a  slow process,
and this is not surprising.

	However,   present    programs    do    little    interesting
introspection.   This is just  a matter of  the undeveloped  state of
artificial  intelligence; programmers  don't yet know  how to  make a
computer program look at itself in a useful way.

#.  %3Consciousness and self-consciousness%1.  In accordance with the
general  approach  of  this  paper,  a  being  is  considered
self-conscious iff it has certain beliefs about itself. However,
we must remember that beliefs are taken as sentences in our language,
and  by ascribing beliefs  we are  not asserting that  the being uses
that language directly or any other language.

	Here is  a hypothesis  arising  from artificial  intelligence
concerning the relation between language and thought.  Imagine a
person or machine  that represents  information internally in  a huge  network.
Each  node of  the  network has  references  to other  nodes  through
relations.    (If the system has  a  variable  collection of
relations, then the relations have to be represented by nodes, and
we get a symmetrical theory if we suppose that each node is connected
to a  set of pairs of other nodes).  We can imagine this structure to
have a long term part and also extremely temporary parts representing
current  %2thoughts%1.    Naturally,  each  being  has  a
its own network depending on  its own experience. A  thought is then  a
temporary  node  currently  being  referenced  by  the  mechanism  of
consciousness.  Its  meaning is determined by its references to other
nodes which  in turn  refer to  yet other  nodes.   Now consider  the
problem of communicating a thought to another being.

	Its full communication would  involve transmitting the entire
network  that  can  be  reached  from  the  given  node, and this would
ordinarily constitute the entire experience of the being.   More than
that,  it would be  necessary to  also communicate the  programs that
that take action on the basis of encountering certain nodes.  Even if
all this could be transmitted, the recipient would still have to find
equivalents  for  the  information  in  terms  of  its  own  network.
Therefore, thoughts  have to  be translated  into  a public  language
before they can be commuunicated.

	A language  is also a network  of associations
and  programs.  However, certain  of the nodes  in this network (more
accurately a %2family%1 of networks, since no two people speak precisely the same
language) are  associated with words  or set phrases.  Sometimes the
translation from thoughts to sentences is easy,
because large  parts of the  private
networks are taken from the public network, and there is an advantage
in preserving the correspondence.  However, the translation is always
approximate (in sense that still lacks a technical definition),
and some  areas of
experience  are  difficult to  translate  at  all.  Sometimes this is for
intrinsic  reasons, and  sometimes because  particular cultures don't
use language in this area.  (It is my impression that cultures differ
in the  extent to which information about  facial appearance that can
be used for recognition is  verbally transmitted).  According to  this
scheme, the "deep structure"  of a publicly expressible thought  is a
node in the public network.  It is translated into the deep structure
of a sentence as a tree  whose terminal nodes are the nodes to  which
words or set  phrases are attached.  This "deep  structure" then must
be translated into a string in a spoken or written language.

	The need to use language to express thought also applies when
we have to ascribe thoughts to other beings, since  we cannot put the
entire network into a single sentence.

→→→→→→→→→→There is more to come here about what ideas are ←←←←←←←←←←←
→→→→→→→→→→needed for self-consciousness.←←←←←←←←←←←←←←←←←←←←←←←←←←←←←

	#. %3Intentions.%1

	We may say that a machine intends to perform an action when it
believes that it will perform the action and it believes that the
action will further a goal.
However, further analysis may show that no such first order definition
in terms of belief adequately describes intentions.  In this case,
we can try a second order definition based on an axiomatization of
a predicate %2I(a,s)%1 meaning that the machine intends the action
⊗a when it is in state ⊗s.

	#. %3Free will%1

	When we program a computer  to  make
choices  intelligently  after determining its options,
examining their consequences, and deciding which
is  most  favorable or most moral or whatever, we must
program it  to  take  an  attitude  towards  its  freedom  of  choice
essentially  isomorphic  to  that  which a human must take to his own.

	We can define whether a particular action
was free or forced relative to a theory
that ascribes beliefs and within which
beings do what they believe will advance their goals.
In such a theory, action is precipitated by a belief of the form
%2I should do X now%1.  We will say that the action was free if
changing the belief to %2I shouldn't do X now%1 would have resulted
in the action not being performed.
This requires that the theory of belief have sufficient Cartesian
product structure so that changing a single belief is defined, but it
doesn't require defining what the state of the world would be if
a single belief were different.

	This isn't the  whole free will story, because  moralists are
also  concerned with whether praise  or blame may be  attributed to a
choice.   The following  considerations would  seem to  apply to  any
attempt to define  the morality of actions in a  way that would apply
to machines:

&. There is unlikely  to be a simple behavioral  definition.  Instead
there would be  a second order definition criticizing predicates that
ascribe morality to actions.


&. The theory must contain at least one axiom of morality that is not
just a statement of physical fact.  Relative to this axiom, judgments
of actions can be factual.

&. The theory of morality will presuppose a theory of belief in which
statements of the form %2"It believed the action would harm someone"%1
are defined.  The theory must ascribe beliefs about others' welfare and
perhaps about the being's own welfare.

&. It might be necessary to consider the  machine as imbedded in some
kind of society in order to ascribe morality to its actions.

&. No  present machines admit  such a belief  structure, and  no such
structure  may be required  to make  a machine with  arbitrarily high
intelligence in the sense of problem-solving ability.

&. It seems  doubtful that morally  judgable machines or  machines to
which rights might legitimately be ascribed are desirable if and when
it becomes possible to make them.

→→→→→→More mental qualities will be discussed.←←←←←←←←←

.SKIP TO COLUMN 1
.ONCE CENTER
.bb OTHER VIEWS ABOUT MIND

→→→→→This section will be written←←←←←
.SKIP TO COLUMN 1
.PORTION NOTES
.bb NOTES

.RECEIVE;

.ITEM←NOTE;
	#. Philosophy and artificial intelligence.  These fields overlap
in the following way:  In order to make a computer program behave
intelligently, its designer must build into it a view of the world
in general, apart from what they include about
particular sciences.  (The skeptic who doubts
whether there is anything to say about the world apart from the
particular sciences should try to write a computer program that can
figure out how to get to Timbuktoo, taking into account not only
the facts about travel in general but also facts about what people
and documents  have what information, and what information will
be required at different stages of the trip and when and how it is
to be obtained.  He will rapidly discover that he is lacking a %2science
of common sense%1, i.e. he will be unable to formally express and
build into his program "what everybody knows".  Maybe philosophy
could be defined as an attempted %2science of common sense%1,
or else the %2science of common sense%1 should be a definite part
of philosophy.)

	Artificial intelligence has a another component in which
philosophers have not studied, namely %2heuristics%1.  Heuristics
is  concerned  with:  given  the  facts  and  a  goal,  how should it
investigate the possibilities and decide what to do.
On the other hand, artificial intelligence is not much concerned
with aesthetics and ethics.

	Not all approaches to philosophy lead to results relevant to
the artificial intelligence problem.  On the face of it, a philosophy
that entailed the view that artificial intelligence was impossible
would be unhelpful, but besides that, taking artificial intelligence
seriously suggests some philosophical points of view.  I am not sure
that all I shall list are required for pursuing the AI goal -
some of them may be just my prejudices - but here they are:

		&. The relation between a world view and the world
should be studied by methods akin to metamathematics in which
systems are studied from the outside.  In metamathematics we study
the relation between a mathematical system and its models.  Philosophy
(or perhaps %2metaphilosophy%1) should study the relation between
world structures and systems within them that seek knowledge.
Just as the metamathematician can use any mathematical methods
in this study and distinguishes the methods he uses form those
being studied, so the philosopher should use all his scientific
knowledge in studying philosphical systems from the outside.

	Thus the question %2"How do I know?"%1 is best answered by studying
%2"How does it know"%1, getting the best answer that the current state
of science and philosophy permits, and then seeing how this answer stands
up to doubts about one's own sources of knowledge.

	&. We  regard  %2metaphysics%1 as  the study  of the  general
structure  of   the  world  and  %2epistemology%1  as  studying  what
knowledge of  the world  can be  had by  an  intelligence with  given
opportunities  to observe  and experiment.   We  need to  distinguish
between  what can  be determined  about the  structure of  humans and
machines  by  scientific   research  over  a   period  of  time   and
experimenting with many individuals, and  what can be learned by in a
particular situation with particular opportunities to observe.   From
the AI point of view, the latter is as important
as the  former, and we suppose that  philosophers would also consider
it part of epistemology.  The possibilities of reductionism  are also
different  for  theoretical  and  everyday epistemology.    We  could
imagine that the rules of everyday epistemology could be deduced from
a knowledge of physics and the structure of the  being and the world,
but  we  can't see  how  one could  avoid  using  mental concepts  in
expressing knowledge actually obtained by the senses.

	&.  It  is  now  accepted that the basic concepts of physical
theories are far removed from observation.  The  human  sense  organs
are  many  levels  of  organization  removed  from quantum mechanical
states, and we have learned to accept the complication this causes in
verifying physical theories. Experience in trying to make intelligent
computer programs suggests that the  basic  concepts  of  the  common
sense  world  are  also complex and not always directly accessible to
observation.   In  particular,  the  common  sense  world  is  not  a
construct  from  sense  data,  but sense data play an important role.
When a man or a computer program sees a dog, we will  need  both  the
relation  between  the  observer and the dog and the relation between
the observer and the brown patch in order to construct a good  theory
of the event.

	&.  In spirit this paper is  materialist, but it is logically
compatible  with some  other philosophies.   Thus  cellular automaton
models of the  physical world may be  supplemented by supposing  that
certain  complex  configurations  interact with  additional  automata
called   souls   that  also   interact   with  each   other.     Such
%2interactionist  dualism%1   won't  meet   emotional  or   spiritual
objections to  materialism, but it does provide a logical  niche for any
empirically  argued  belief in  telepathy, communication with the dead
and  other   psychic phenomena.
A person who believed the alleged evidence for such phenomena and
still wanted a scientific explanation could model his beliefs
with auxiliary automata.

.SKIP TO COLUMN 1
.bb REFERENCES

%3Carnap, Rudolf%1 (1956), %2Meaning and Necessity%1, University of Chicago
Press.

%3McCarthy, J. and Hayes, P.J.%1 (1969) Some Philosophical Problems from
the Standpoint of Artificial Intelligence. %2Machine Intelligence 4%1,
pp. 463-502 (eds Meltzer, B. and Michie, D.). Edinburgh: Edinburgh
University Press.

→→→→→→→→More references will be supplied←←←←←←←←
.BEGIN VERBATIM

John McCarthy
Artificial Intelligence Laboratory
Stanford University
Stanford, California 94305
.END